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[57] ABSTRACT 

A method of processing images for constructing a tar- 
get image (Io) from adjacent images having a fixed 
frame line and referred to as source images (II, . . . , Ii, 
Ij, . . . , In), the source and target images having substan- 
tially common view points. This method includes the 
steps of: digitizing the images, determining, for one of 
the pixels of the target image (Io), the address (Aq) of a 
corresponding point in one of all source images (Ij), 
deterrnining the luminance value (F) at this correspond- 
ing point, assigning the luminance value (F) of this 
corresponding pixel to the initial pixel in the target 
image (Io), and repeating these steps for each pixel of 
the target image (Io). A device for reforming this 
method includes a system of n fixed real cameras (CI, . 
. . , Cn) which provide n adjacent source images (II, . . 
. , In) covering a wide-angle field of view and which 
have common view points (P), and an image recon- 
struction system (100) simulating a mobile camera re- 
ferred to as target image (Co) for providing a sub-image 
referred to as target image (Io) of the wide-angle field of 
view, and constructed on the basis of source images 
having the same view point (P). 

17 Claims, 6 Drawing Sheets 
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IMAGE PROCESSING METHOD AND DEVICE 
FOR CONSTRUCTING AN IMAGE FROM 
ADJACENT IMAGES 

BACKGROUND OF THE INVENTION 
1. Held of the Invention 

The invention relates to a method of processing im- 
ages for constructing a target image from adjacent im- 
ages having a fixed frame line and referred to as source 
images, said source and target images having substan- 
tially common view points. 

The invention also relates to an image processing 
device comprising: 
a system of n fixed real cameras arranged in such a 
way that their individual fields of view merge so as 
to form a single wide-angle field of view for obser- 
vation of a panoramic scene, 
an image construction system simulating a mobile, 
virtual camera continuously scanning the pan- 
oramic scene so as to form a sub-image referred to 
as target image corresponding to an arbitrary sec- 
tion of the wide-angle field of view and con- 
structed from adjacent source images furnished by 
the n real cameras, said virtual camera having a 25 
view point which is common with or close to that 
of the real cameras. 
The invention is used in the field of telemonitoring or 
in the field of television where shots covering large 
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include a displacement perpendicular to the scan; its 
localization only includes displacements parallel to this 
scan. The formation of the sub-image does not include 
the zoom effect with respect to the composite image, i.e. 
the focal change of the sub-image with respect to the 
focal length of the image pick-up cameras. 

The image processing station thus comprises means 
for constructing the selected video sub-image line after 
line. These means essentially include a circuit for con- 
trolling the synchronization of the video signals from 
the different cameras. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to provide a 
device which is capable of simulating a mobile camera 
scanning the wide-angle field of view covered by the n 
fixed cameras whose fields of view merge. 

A particular object of the present invention is to 
provide such a device simulating a camera which is 
provided with all the facilities of a real existing mobile 
camera, i.e. from a stationary observer, possibilities of 
horizontal angular displacements towards the left or the 
right of a panoramic scene to be observed or to be moni- 
tored, possibilites of vertical angular displacements to 
the top or the bottom of this scene, possibilities of rota- 
tion and also possibilities of zooming in on a part of the 
surface area of this scene. 
This object is achieved by performing a method of 
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events. The invention is also used in the field of automo 
bile construction for realizing peripheral and panoramic 
rear-view means without a blind angle. 

2. Description of the Related Art 

An image processing device is known from Patent 
Application WO 92-14341, corresponding to U.S. Pat. 
No. 5,187,571. This document describes an image pro- 
cessing system for television. This device comprises a 
transmitter station including a plurality of fixed cameras 
arranged adjacent to each other so that their fields of 40 
view merge and form a wide-angle field of view. This 
system also comprises a processing station including 
means for generating a composite video signal of the 
overall image corresponding to the wide-angle field of 
view, and means for selecting a sub-image from this 45 
composite image. This system also comprises means, 
such as a monitor, for displaying this sub-image. This 
sub-image corresponds to a field of view having an 
angle which is smaller than that of the composite image 
and is referred to as sub-section of the wide-angle field 
of view. 

This image processing device is solely suitable for 
conventional television systems in which the image is 
formed line by line by means of a scanning beam. 

The processing station enables a user to select the 55 
sub-section of the wide-angle field of view. The corre- 
sponding sub-image has the same dimension as the 
image furnished by an individual camera. The user se- 
lects this sub-image by varying the starting point of the 
scan with respect to the composite image correspond- 
ing to the wide-angle field of view. The wide-angle field 
of view has an axis which is parallel to the video scan, 
with the result that the starting point for the video scan 
of the sub-image may be displaced arbitrarily and con- 
tinuously parallel to this axis. 

The angle of the field of view to which the sub-image 
corresponds may be smaller than that of a real camera. 
However, the localization of the sub-image does not 
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to as source images, said source and target images hav- 
ing substantially common view points, characterized in 
that the method comprises the steps of: 
digitizing the images, 

determining, for one of the pixels of the target image, 
the address of a corresponding point in one of all 
source images, 

determining the luminance value at this correspond- 
ing point, 

assigning the luminance value of this corresponding 
pixel to the initial pixel in the target image, 

repeating these steps for each pixel of the target im- 
age. 

According to the invention, for performing this 
method, an image processing device is also proposed, 
which device includes: 

a system of n fixed real cameras arranged in such a 
way that their individual fields of view merge so as 
to form a single wide-angle field of view for obser- 
vation of a panoramic scene, 

an image construction system simulating a mobile, 
virtual camera continuously scanning the pan- 
oramic scene so as to form a sub-image referred to 
as target image corresponding to an arbitrary sec- 
tion of the wide-angle field of view and con- 
structed from adjacent source images furnished by 
the n real cameras, said virtual camera having a 
view point which is common with or close to that 
of the real cameras, characterized in that this image 
processing device is a digital device and in that the 
system (100) for constructing the target image Io 
includes: 

an address computer for causing a point at an address 
in one of the source images to correspond to a pixel 
address in the target image, 

means for computing the luminance value of the point 
at the address found in the source image and for 
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assigning this luminance value to the initial pixel at 

the address in the target image. 
Thus, the device according to the invention provides 
the possibility of constructing a target image like the 
one furnished by a supposed camera which is being 
displaced in a continuous manner, this target image is 
formed from several adjacent source images each pro- 
vided by one camera from a group of cameras arranged 
in a fixed manner with respect to the scene to be ob- 
served, and, based on this construction, this device may 
furnish, by way of display on the screen, or by way of 
recording: 

either a sequential image-by-image read-out of parti- 
tions of the observed scene, possibly with a zoom 
effect, 

or a continuous read-out by scanning the scene ob- 
served with the sight and azimuth effect or with 
rotation. 

In a particular embodiment, this device is character- 
ized in that the target image reconstruction system com- 
prises: 

first means for storing the parameters relating to the 
virtual camera for supplying the address computer 
with the scale factor and the orientation of the 
optica] axis of the virtual camera in a fixed ortho- 
normal landmark which is independent of the cam- 
eras, i.e. the azimuth angle, the angle of sight and 
the angle of rotation; 
second means for storing the parameters relating to 
the real cameras for supplying the address com- 
puter with the scale factor and the orientation of 
the optical axis of each real camera, Le. their azi- 
muth angle, the angle of sight and the angle of 
rotation in said fixed landmark; 
an address generator for generating, pixel by pixel, 
the addresses (Ao) of the pixels of the target image 
so as to cover the entire target image, the address 
computer determining the particular source image 
and the point at the address (Aq) in this source 
image, which corresponds to each pixel of the 40 
target image, on the basis of the parameters of the 
virtual camera and the real cameras. 
Another technical problem is posed by the construc- 
tion of the target image. It is supposed that a plurality of 
cameras is arranged adjacent to one another and that no 45 
zone of the panoramic scene to be constructed is be- 
yond the field covered by each camera: it is thus sup- 
posed that all the data for constructing the target image 
are provided. Nevertheless, at each boundary between 
the cameras, where an image from one camera passes to 
another image of an adjacent camera, the viewing angle 
difference between these two cameras for two adjacent 
zones of the scene recorded by these two different cam- 
eras causes great distortions of the image. The result is 
that the partitions which are realized on and at both 55 
sides of the two zones of the scene recorded by two 
different cameras are very difficult to display and com- 
pletely lack precision. 
It is another object of the invention to provide a 
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second means for constructing models (MCl-MCn) 
of the real cameras with a projection via the view 
point and with corrections of distortions and per- 
spective faults. 

In a particular embodiment, this device is character- 
ized in that the address computer comprises: 

first means for computing the geometrical transform 
for applying a geometrical transform referred to as 
inverse "perspective transform" (H<>- 4 ) to each 
pixel at an address (Ao) of the image of the virtual 
camera, in which transform the model (MCo) of 
the virtual camera provided by the first construc- 
tion means and the parameters for the azimuth 
angle, the angle of sight, the angle of rotation and 
the scale factor of this virtual camera provided by 
the first storage means are taken into account for 
determining, on the basis of this inverse perspective 
transform (Ho -4 ), the positioning in said landmark 
of the light ray passing through this pixel and the 
view point, 

means for storing the position of the light ray ob- 
tained by the inverse perspective transform (H 0 - 4 ), 
means for selecting the particular source image tra- 
. versed by this light ray, 

second means for computing the geometrical trans- 
form for applying a geometrical transform referred 
to as "direct perspective transform" (Hl-Hn) to 
this light ray in said landmark, in which transform 
the models of the real cameras provided by the 
second construction means, the parameters for the 
azimuth angle, the angle of sight, the angle of rota- 
tion and the scale factor of the corresponding real 
camera provided by the second storage means are 
taken into account, 
and storage means for supplying, on the basis of this 
direct perspective transform (Hl-Hn), the address 
(Aq) in the particular source image which corre- 
sponds to the light ray and thus to the pixel of the 
address (Ao) in the target image. 
With this device, the user who monitors a panoramic 
scene exactly obtains the same convenience of use and 
the same service as a user of a mobile camera with zoom 
and mechanical means for realizing the variation of the 
orientation of the optical axis, i.e., for realizing varia- 
tions of sight and azimuth, as well as rotations around 
the optical axis of the camera. The advantage is that the 
mechanical means are not necessary. These mechanical 
means, which include mechanical motors for rotating 
the azimuth angle and the angle of sight and a motor for 
zoom control always have drawbacks: first, they may 
get blocked and then the generated displacements are 
very slow. Moreover, they are very expensive. As they 
are most frequently installed externally, they will rap- 
idly degrade because of poor weather conditions. The 
electronic image processing means according to the 
invention obviate all these drawbacks because they are 
very precise, reliable, very rapid and easy to control. 
Moreover, they may be installed internally and thus be 



construction of the target image whose image distortion 60 sheltered from bad weather. The electronic means are 



at the boundary between two cameras is corrected so 
that this (these) boundary(ies) is (are) completely invisi- 
ble to the user. 

This object is achieved by means of an image process- 
ing device as described hereinbefore, which is charac- 
terized in that the address computer comprises: 

first means for constructing a model (MCo) of the 
virtual camera with a projection via the view point, 
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also easily programmable for an automatic function. 
Finally, they are less costly than the mechanical means. 

With the means according to the invention, the user 
thus obtains an image which is free from distortions and 
has a greater precision and an easier way of carrying out 
the sighting operations than with mechanical means. 
Moreover, a panoramic scene of a larger field may be 
observed because fields of 180° or even 360°, dependent 
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on the number of cameras used, can be observed. The FIG. 5A illustrates the models of the real and virtual 
operations can also be easily programmed. cameras; 

Great progress is achieved as regards surveillance. As FIG. 5B illustrates, in projection on the horizontal 
for realizing panoramic rear-view means for automo plane of the landmark, the perspective and distortion 
biles, this progress is also very important. 5 effects on the positions of the corresponding points 

The fact that several cameras are used for acquiring having the same luminance in the target image and in 
data which are necessary for constructing the target the source image traversed by the same light ray passing 
image is not a disadvantage, because such an assembly through these points; 

of fixed CCD cameras has become less difficult to nan- FIG. 6 shows, in the form of functional blocks, the 
die than the mechanical devices for varying the sight, 10 address computer which computes the address of the 
azimuth and rotation, as well as the zoom for a single point in the source image corresponding to a pixel at an 
real mobile camera. address in the target image; 

In a particular embodiment, this system is character- FIG. 7A shows a first digital source image formed by 
ized in that the means for determining the luminance a first real fixed camera and FIG. 7B shows a second 
comprise: 15 source image formed by a second real fixed camera 

an interpolator for computing a most probable value adjacent to the first camera; 
of a luminance function (F) at the address (Aq) FIG. 7C shows a digital target image reconstructed in 
found by the address computer in the source image the same manner as in the case of FIG. IF showing the 
furnished by the selection means; distortion and perspective faults between the first target 

third storage means for assigning the luminance value 20 image part constructed on the basis of the first source 
(F) corresponding to the point at the address (Aq) image and the second target image part constructed on 
found in the source image to the initial pixel in the the basis of the second source image; and 
target image at the address (Ao) furnished by the FIG. 7D shows the digital target image of FIG. 7C 
address generator, and in that the system for recon- after treatment by the image processing device, in 
structing the target image also comprises: 25 which the distortion and perspective faults have been 

an interface for enabling a user to define the parame- eliminated. 

ters of the virtual camera, which parameters in- _ _ 

elude the scale factor and the orientation of the DESCRIPTION OF THE PREFERRED 

optical axis. EMBODIMENT 
These and other aspects of the invention will be ap- 30 I/The image pick-up system, 
parent from and elucidated with reference to the em- FIG. 1G shows a possible arrangement of several real 
bodiments described hereinafter. fixed cameras for recording the data relating to a scene 

BRIEF DESCRIPTION OF THE DRAWINGS ^rAM^^^lT^ P ^T r Trf " 

corded with three fixed cameras CI, C2, C3. The cam- 
In the drawings 35 eras have such optical fields that, absolutely, all the 
FIG. 1 A is a plan view showing the traces of the details of the panoramic scene are recorded by the one 
different image planes in the horizontal plane of the or the other camera so that no object under surveillance 
landmark in the case where the real cameras have image is left out. The cameras are arranged to have a common 
planes which are perpendicular to this horizontal plane; view point P or very close view points. 

FIG. IB shows the landmark Px, Py, Pz viewed in 40 The axes PZ1, PZ2, PZ3 represent the optical axes of 
projection in the horizontal plane; the cameras CI, C2, C3, respectively, and the points 01, 

FIG. 1C is an elevational view of a source image 02, 03 represent the geometrical centers of the images 
plane with its particular system of coordinate axes; II, 12, 13, respectively, in the image planes on the opti- 

FIG. ID is an elevational view of the target image cal axes, 
plane with its particular system of coordinate axes; 45 A horizontal surveillance through 360° can be carried 
FIG. IE represents the effect of limiting a section of out by suitably arranging 6 fixed cameras. However, a 
the wide-angle field of view of two adjacent real cam- vertical surveillance or a surveillance in both directions 
eras by means of parameters chosen by the user for the may also be carried out. Those skilled in the art will be 
virtual camera for constructing a sub-image of a pan- able to realize any type of system for observation of a 
oramic scene; 50 panoramic scene so that a more detailed description of 

FIG. IF shows the target image constructed by the the various mutual arrangements of the fixed cameras is 
virtual camera defined by these parameters, this target not necessary. 

image being composed of a first part of an image con- With reference to FIG. 1A, the image pick-up device 
structed on the basis of the source image furnished by comprises a plurality of n fixed real cameras having 
the first of the two real cameras and of a second image 55 known and fixed focal lengths and being arranged adja- 
part constructed on the basis of the source image fur- cent one another so that their individual fields of view 
nished by the second of these cameras; merge to cover a wide-angle field of view. The n adja- 

FIG. 1G shows an arrangement of three adjacent real cent fixed cameras furnish n adjacent fixed images so 
cameras for covering a field of view of 180*; that this image pick-up device can monitor a panoramic 

FIG. 2 shows, in the form of functional blocks, the 60 scene. The cameras have such optical fields that all the 
image processing device with the system for construct- details of the panoramic scene are recorded by the one 
ing the target image, the real cameras, the user interface or the other camera so that no object under surveillance 
and the system for displaying the target image; is left out. 

FIG. 3 shows the image processing device in the form To obtain this result, these n adjacent fixed cameras 
of functional blocks in greater detail than in FIG. 2; 65 are also arranged in such a way that their optical centers 

FIG. 4 illustrates the computation of a value of a P, referred to as view points coincide. The view point 
luminance function relative to an address in a source of a camera is defined as the point at which each ray 
image; emitted from a luminous source and passing through 
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this point traverses the optical system of the camera 
without any deviation. 

The view points of the n cameras need not coincide 
physically. However, it will hereinafter be assumed that 
the condition of coincidence is fulfilled sufficiently if 
the distance separating each of these view points is 
small as regards their distance to the filmed panoramic 
scene, for example, if their respective distance is 5 cm or 
10 cm and the distance to the panoramic scene is 5 m. 
The condition of coincidence is thus estimated to be 
fulfilled if the ratio between these distances is of the 
order of or is more than 50 and, according to the inven- 
tion, it is not necessary to use costly optical mirror 
systems which are difficult to adjust for achieving a 
strict coincidence of the view points. 
II/Fonnation of the images by the cameras. 

It is an object of the invention to provide a system for 
reconstructing a digital image which simulates a mobile 
camera which, with the settings selected by a user, is 
capable of furnishing a digital image of any part, or 20 
sub-image, of the panoramic scene recorded by the n 
fixed cameras. 

The n cameras are numbered CI, . . . , Ci, Cj, . . . , Cn 
supplying digital source images II, ... , Ii, Ij, . . . , In, 
respectively. For example, the source images Ii and Ij 
formed by two adjacent fixed real cameras Ci and Cj 
will be considered hereinafter. 

These fixed real cameras Ci and Cj form respective 
images of the panoramic scene in adjacent source image 
planes Ii and Ij. In FIG. 1A the axes Pzi and Pzj passing 
through the geometrical centers Oi and Oj of the source 
images Ii and Ij, respectively, represent the optical axes 
of the fixed real cameras Ci and Cj. 

With reference to FIG. IB, a landmark Px, Py, Pz of 
orthogonal axes is defined in which the axes Px and Pz 35 
are horizontal and the axis Py is vertical. 

The source images, such as the images Ii and Ij, are 
numbered and each pixel m of these images is marked 
by way of its coordinates in the image plane. As is 
shown in FIG. 1C, a mark of rectangular coordinates 40 
(OiXi, OiYi) and (QjXj, OjYj) are defined in each image 
plane in which the axes OiXi, or OjXj are horizontal, 
i.e., in the plane of the landmark Px, Pz. The image 
planes defined by (OiXi, OiYi) and (OjXj, OjYj) are 
perpendicular to the optical axes Pzi and Pzj and have 45 
respective geometrical centers Oi and Oj. 

Once these individual marks relating to each image 
plane of the cameras are established, these fixed source 
image planes may be related to the landmark by means 
of: 

their azimuth angle (or pan angle) 9i, Gj, 

their angle of sight (or tilt angle) <f>i, <f>j. 

The azimuth angle 0i or Oj is the angle forming a 
vertical plane containing the optical axis PZi or PZj 
with the horizontal axis Pz of the landmark. Thus, this 55 
is a horizontal angle of rotation about the vertical axis 

Py- 

The angle of sight <f>i or <j>j is the angle formed by the 
optical axis PZi PZj with the horizontal plane (Px, Pz). 
Thus, this is a vertical angle of rotation about a horizon- 60 
tal axis, the axis OiXi or OjXj of each image plane. 

For reasons of simplicity, it has hereinafter been as- 
sumed, by way of example with reference to FIG. 1A, 
that the source image planes Ii, Ij furnished by the fixed 
cameras Ci, Cj are vertical, i.e. their angles of sight <f>i, 65 
4>j are zero. 

For similar reasons of simplicity, the same reference 
in FIG. 1A denotes the trace of the planes and the axes 
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and the corresponding planes and axes for both the 
source images and for the target image described here- 
inafter. 

FIG. 1A, which is a diagrammatic plan view of the 
images formed, thus only shows the traces It and Ij of 
the fixed source image planes represented by segments 
in the horizontal plane Px, Pz. 

FIG. IE shows, for example, the contiguous images Ii 
and Ij of the panoramic scene, furnished by two adja- 
cent fixed cameras Ci and Cj. In FIG. IE, both images 
Ii and Ij are projected in the same plane for the purpose 
of simplicity, whereas in reality these images constitute 
an angle between them which is equal to that of the 
optical axes of the fixed cameras. In these images, the 
user may choose to observe any sub-image bounded by 
the line Jo more or less to the left or to the right, more 
or less to the top or to the bottom with the same magni- 
fication as the fixed cameras or with a larger magnifica- 
tion, or possibly with a smaller magnification. 

The simulated mobile camera is capable of construct- 
ing a target image Io from parts of the source image Si, 
Sj bounded by the line Jo in FIG. IE. This camera, 
denoted by Co hereinafter, is referred to as the virtual 
camera because it simulates a camera which does not 
really exist. Evidently, this simulated mobile camera is 
not limited to scanning the two images Ii, Ij. It may scan 
all the source images from II to In. 

This virtual camera Co can be defined in the same 
manner as the fixed real camera by means of: 

its azimuth angle Go 

its angle of sight <f>o 

its angle of rotation \J*o 

and its magnification (zoom effect) defined by its 
focal length POo, and denoted as zo, with its view 
point P being common with the view points P of 
the fixed real cameras, while Oo is the geometrical 
center of the target image Io. The view point of the 
virtual camera is common with the approximate 
view point as defined above for the real cameras. 

FIG. 1A shows the trace denoted by Io of the image 
plane of the virtual camera in the horizontal plane and 
its optical axis PZo passing through the geometrical 
centre Oo of the target image Io. 

In the definition of this mobile virtual camera Co, the 
azimuth angle So is the angle made by the vertical plane 
containing its optical axis PZo with the horizontal axis 
Pz of the landmark; the angle of sight <J>o is the angle 
made by its optical axis PZo with the horizontal plane 
Px, Pz of the landmark; its angle ifro is the angle of 
rotation of the virtual camera about its own optical axis, 
the latter being fixed; and finally, its focal length POo is 
variable so that the magnification of this target image 
with respect to that of the source images can be 
changed (zoom effect). 

By varying the azimuth angle Go and the angle of 
sight <f>o, the angle of rotation i/ro and the focal length 
POo, the virtual camera is entirely similar to a mobile 
camera which scans the wide-angle field of view 
formed by the merged fields of view of the different 
fixed real cameras CI to Cn. 

It is to be noted that the virtual camera Co can view 
a small part (or subsection) bounded by Jo of the wide- 
angle field of view and by realizing a magnified image 
Io, for example, of the same final dimension as each of 
the images II, . . . , In furnished by each real camera CI, 
, . . , Cn by varying its variable focal length POo. 

It is also to be noted that the displacement of the field 
of view of the virtual camera Co may be continuous and 
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arbitrary; this field of view corresponding to Jo may be 
on or at both sides of the two parts (Si, Sj) of the contig- 
uous images Ii and Ij at LO, furnished by two adjacent 
cameras Ci and Cj. 

In this case, the image Io constructed by the virtual 5 
camera Co contains two different image parts, one part 
Ioi being constructed on the basis of information Si in 
the digital image Ii and the other part Ioj being con- 
structed on the basis of information Sj in the digital 
image Ij. In FIG. 1A, Ioi and Ioj represent the traces of 10 
the target images Ioi and Ioj in the horizontal plane. 

Likewise as for the real images, a mark of rectangular 
coordinates (Oo Xo, Oo Yo) will now be defined with 
reference to FIG. ID in the digital target image plane 
Io, in which mark the axis Oo Xo is horizontal, i.e. in 15 
the horizontal plane of the landmark Px, Pz. The pixel 
Oo is the geometrical, center of the target image Io and 
is also situated on the optical axis PZo of the virtual 
camera Co. Each pixel m' of the target image plane Io 
is thus marked by its coordinates in this system of rect- 20 
angular axes (Oo Xo, Oo Yo). 

A framework of the target image is also defined, i.e., 
its dimensions in numbers of pixels in the horizontal 
direction and in numbers of lines in the vertical direc- 
tion. 25 
Ill/The method of constructing the target image from 
source images. 

The digital image processing means according to the 
invention are used for constructing the "target image" 
furnished by the virtual camera from "source images" 30 
furnished by the fixed real cameras. 

As stated hereinbefore, a technical problem may 
occur in the construction of the target image Io because 
the images Ii and Ij furnished by the adjacent real cam- 
eras and represented in the form of digital landscape 35 
images in FIGS. 7A and 7B show distortions which do 
not correctly blend: particularly certain parts of the 
straight lines are curved, while the vertical lines are not 
vertical, etc. As a result, when they are joined, these 
lines may be cut instead of being in alignment. More- 40 
over, the parts of the source images at both sides of the 
joint originate from panoramic scene parts which are 
viewed under different perspectives. The result is that 
the parts of the target images Ioi and Ioj are poorly 
joined, as is apparent in the digital target image shown 45 
by way of example in FIG. 7C and formed by the simple 
juxtaposition of the parts of the digital images shown in 
FIGS. 7A and 7B. 

The present invention proposes a method and means 
for eliminating these faults and provides a reconstructed 50 
digital image which does not have distortion and per- 
spective faults and whose parts Ioi, Ioj from which it is 
composed are joined in such a manner that the straight 
boundary is invisible to the observer and the user. 

FIGS. 7D shows the digital landscape image of FIG. 55 
7C in which the distortion and perspective faults have 
been corrected by the means according to the invention. 

The general method of constructing the source image 
comprises different steps which are carried out by the 
signal processing device described hereinafter. 60 

This method first comprises a step in which: 

a point m at an address Aq in a source image is made 
to correspond to each pixel m' having an address 
Ao in the target image Io, which address is defined 
by: 65 

the reference or index of the real camera furnishing 
the source image; 

the address Aq of the point m in this source image. 
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This method comprises a second step in which: 

the most probable luminance value is evaluated at 
said point m in the source image, 

subsequently this luminance value is assigned to the 
pixel m' in the target image. 

These steps are carried out for all the pixels m' in the 
target image Io. 

The processing means may give the constructed tar- 
get image all the qualities of an image obtained by an 
observer using a conventional mobile camera: 

absence of distortions, adjustment of perspectives, 

absence of straight interrupting lines at the boundary 
between two or more adjacent images. 
The problem thus is to render these straight boundaries 
invisible. 

IV/Essential elements of the image processing device. 

FIG. 2 shows the different elements of the image 
processing device according to the invention in the 
form of functional blocks. 

The blocks C 1, Ci, Cj, . . . , Cn represent the n fixed 
real cameras whose outputs supply the n source images 
denoted by II, . . . , Ii, Ij, . . . , In. 

Each real camera has pick-up parameters correspond- 
ing to its orientation with respect to the scene, as well as 
a scale factor. These parameters should be precisely 
known. 

The block 1 shows a control system which enables 
the user to choose and display parameters relative to the 
orientation and to a scale factor of the virtual camera 
Co. 

The block 100 is an image reconstruction system 
which allows computation, on the basis of source im- 
ages II, ... , In, which are provided by the real cameras 
Cl f . . . , Cn, of the image referred to as "target image" 
Io given by the virtual camera Co which is oriented and 
adjusted by means of the parameters defined by the 
control system 1, which virtual camera Co is arranged 
in such a way that its view point is common with or 
close to the view point P of the real cameras. 

The block 10 corresponds to display means, or re- 
cording means which may be, inter alia a digital televi- 
sion monitor for real time display and/or a video re- 
corder for recording on video tapes, etc. 

It is to be noted that the real cameras may furnish 
analog data. In this case A/D conversion modules (not 
shown) are used for forming the digital source images. 

Advantageously, those skilled in the art may choose 
CCD cameras (Charge-Coupled Device). This type of 
camera is easy to handle, lightweight, robust and reli- 
able. It has a very good resolution and, in turns of tech- 
nical evolution, it may further be improved in the fu- 
ture. 

V/Detailed elements of the image processing device. 

FIG. 3 shows a diagram of functional blocks corre- 
sponding to the essential means for realizing the pro- 
cessing operation. 

These means principally comprise: 

a user interface 2, 

an address computer 200 including a source image 

selector 205; 
an interpolator 112, 

an address generator 113 which, pixel by pixel, gener- 
ates the addresses Ao of the pixels of the target 
image Io so as to cover the entire target image Io, 

first means 210 for storing the parameters (|>so, 6o, ^o 
and zo (the scale factor) of the virtual camera Co, 
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second means 21.1 to 21.n for storing the parameters 
<J>1 to (Jm, 61 to On, \fm to tyn and zl to zn (the scale 
factors of the real cameras CI to Cn), respectively, 

third storage means 117. 

For each pixel m' of the target image Io to be con- 5 
structed, an address Ao is stored in the block 113 for 
generating the addresses in the target image Io, As a 
function of the settings chosen by the user, which set- 
tings are entered into the control block 1 by means of a 
user-interface 2 and which are stored in the form of 10 
parameters cf>o, Go, tyo and zo of the virtual camera in 
the block 210, and as a function of the parameters <J>i, 9i, 
\jd and zj (j representing the index of all the cameras 
from 1 to n) of the real cameras which are stored in the 
blocks 21.1, 21 j, 21. n, the address computer 200 applies 15 
the index j (among all the indices 1 to n) of the real 
camera Cj which is capable of supplying the image Ij 
with the corresponding data to the address Ao of the 
pixel m' of the target image Io. 

Subsequently, the image selector 205 selects this real 20 
camera Cj among the n real cameras and provides the 
corresponding source image Ij in which these data to be 
processed are found. 

On the basis of the same elements, the address com- 
puter 200 computes the corresponding address Aq of 25 
the point m in this selected source image Ij, and this via 
different perspective transforms performed by the user 
at the interface 2. 

Conformity between a pixel m' at a given address Ao 
of the target image Io and a point m at a computed 30 
address Aq of a source image Ii is thus established. 

Subsequently, the luminance value at the point m at 
the address Aq in the source image Ij is determined in . 
order to be assigned to the corresponding pixel m' at the 
address Ao in the target image Io. This is repeated for 35 
all the pixels of the target image Io. 

Whereas the address Ao in the target image Io is 
really that of a pixel m', the address Aq in the source 
image Ii is exactly a computed address so that, gener- 
ally, this address found does not coincide with a pixel, 40 
but with a point m between the pixels. The luminance of 
this point m should thus also be computed. 

FIG. 4 illustrates the problem which occurs when the 
address Aq does not coincide with any pixel of a source 
image Ii. 45 

This problem is dealt with by the interpolator 112. 

The interpolator 112 takes the light intensity into 
account} i.e., for example, the grey levels or a value of 
the luminance function of the pixels proximate to the 
address Aq for computing the light intensity to be as- 50 
signed to the address Aq in question. By way of interpo- 
lation, the block 112 computes the intensity which is 
most likely to be assigned to the address Aq. 

There are several possible interpolation functions 
which are more or less complex and which, in accor- 55 
dance with their complexity, yield more or less finely 
adjusted results. 

A very simple interpolation function is the bilinear 
interpolation illustrated in FIG. 4. The address Aq com- 
prises a whole part and a fractional part. For this reason, 60 
the address Aq is situated between four neighbouring 
pixels ml, m2, m3, m4. The fractional part of the ad- 
dress is represented by ax positively counted from ml 
to m2 and by ay positively counted from ml to m3. The 
distance between ml and m3 and between ml and m2 is 65 
1 pixel. 

The intensity F or the value of the luminance func- 
tion at the address Aq is computed on the basis of the 
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values of Ax, Ay and the intensities (or values of the 
luminance functions) of the neighboring pixels ml, m2, 
ra3, m4, being Fl, F2, F3, F4, respectively, 

F(Aj) = Fl(l - Ax)(l - Ay) + F2 Ax(l- 
-Ay)+F3(l-Ax)Ay+F4Ax.Ay 

The intensity F(aq) computed by the interpolator at 
the address Aq in the source images is subsequently 
attributed as the intensity (or value of the luminance 
function) to the pixel m' situated at the initial address 
Ao in the target image Io and stored in the block 117. 
VI/Description of the address computer 200 

The address computer 200 will hereinafter be de- 
scribed in detail with reference to FIG. 6. 

The address computer 200 first takes the parameters 
of the virtual and real cameras as defined hereinbefore 
into account. 

The parameters <f>o, 9o, *f*o and zo of the virtual 
camera are available in the storage module 210; the 
parameters <f>l to <f>n, 91 to 9n, iffl to ifm and zl to zn of 
the real cameras are available in the storage modules 
21.1 to 2tn, respectively. 

A definition of the dimensions of the images in num- 
bers of pixels expressed by width X height (in pixels) has 
been determined. The scale factors zo or zi are ex- 
pressed in pixels/millimeter. The angles are expressed, 
for example in degrees. 

For the virtual camera, these parameters <f>o, 9o, ifo, 
zo are computed on the basis of the orientation or posi- 
tioning of the optical axis of the virtual camera, the 
positioning of its view point P and the image center Oo 
and on the basis of the scale factor, which values are 
chosen by the user and entered at the interface 2. 

For the real cameras, the parameters <f>l to <f>n, ©1 to 
9n, $1 to ijm the focal lengths POl to POn, the scale 
factors zl to zn and the centers Ol to On of the digital 
image are determined in a very precise manner on the 
basis of very accurate calibrations and measurements of 
the position of the point P and the centers Ol to On of 
each camera, its scale factor and the orientation or posi- 
tioning of its optical axis. 

The address computer 200 also comprises means 201 
and 206 for constructing a model of the virtual camera 
Co and a model of the real cameras CI, C2, . , . , C3, . 
..,Cn. 

FIG. 5A illustrates the principle of forming a model 
for the virtual camera Co and the real cameras CI, C2, 
. . . , Cn. The projection model shown in FIG. 5A is 
based on a conical projection through a point in accor- 
dance with which all the optical rays pass through the 
single view point P (the pin-hole model). The view 
point P is supposed to be common for all the real cam- 
eras or the virtual camera under the conditions as stated 
hereinbefore. 

The landmark Px, Py, Pz and the individual mark of 
the virtual camera are defined in the construction mod- 
ule 201; the landmark Px, Py, Pz and the individual 
marks of each of the real cameras CI to Cn are defined 
in the construction module 206. 

Let it be assumed, with reference to FIG. 5A, that M 
is an object point situated in a scene to be monitored. 
The point m', corresponding to M in the image plane of 
the virtual camera Co, is situated at the intersection 
between the straight line PM and the image plane Io 
(Ooxo, Ooyo) of the camera Co. 

It is to be noted that the model of FIG. 5A is directly 
valid for the virtual camera which is an ideal camera 
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without distortions. As a real camera is used which is 
provided, for example with a wide-angle aperture ob- 
jective, there are distortions in the image which must be 
corrected. 

FIG. SB illustrates this problem. This FIG. 5B shows 5 
the traces of the planes of the target image Io and the 
source image Ij shown similarly as in FIG. 1A. 

The problem to be solved for constructing the target 
image Io on the basis of source images II to In is to 
determine at which address Aq the point m which cor- 10 
responds exactly to the pixel m' of the address Ao can 
be found. In fact, it is attempted to determine the inten- 
sity which exists in the source image at this point m so 
as to be able to assign it to the pixel m' in the target 
image. 15 

This determination is not simple because the distor- 
tion and perspective faults influence the real cameras. 

Similarly as in FIG. 5A, FIG. SB shows that the pixel 
m' of the address Ao is present at the intersection be- 
tween the light ray PM and the target image Io, PM 20 
being the ray which joins the view point P with a point 
M of the scene to be observed because in this ideal 
camera each ray passing through the view point P tra- 
verses the image plane without any deviation. 

This ray PM intersects the plane of the source image 25 
Ij at a point m'. The camera Cj will thus be the real 
camera selected to provide the data which are neces- 
sary for constructing this part of the target image. 

However, on the other hand, it is not true that the 
point m' at the intersection between the same ray PM 30 
and the plane of the source image Ij is the point corre- 
sponding to m' in Ij which is capable of supplying the 
intensity value of F to be assigned to this point m\ 

The fact should be taken into account that the camera 
Cj, as every camera, has an optical objective which 35 
forms a real image on a substrate provided with a CCD 
matrix; this real image is subsequently transformed by 
an A/D conversion module for supplying the digital 
source image Ij which is then processed by the digital 
image processing device according to the invention. 40 
Due to the fact that it is necessary to use a system of 
objective lenses, it may be considered, as shown dia- 
grammatically in FIG. SB, that the light ray MP is 
deviated by the distortion field DF shown in broken 
lines, with the result that the point corresponding to m' 45 
is not the point m* situated on MP but a point m situated 
at a small distance from m* in the plane of the source 
image Ij. 

With the data relating to the landmark and the indi- 
vidual marks, the construction module 201 builds a 50 
model MCo of the virtual camera on the conical projec- 
tion model represented in FIG. 5A in which each ray 
PM in the three-dimensional space corresponds to a 
pixel m' in the two-dimensional space constituted by the 
plane of the target image Io, because the virtual camera 55 
Co is considered to be an ideal camera. 

On the other hand, with the data relating to the land- 
mark and to the individual marks, the construction 
module 206 builds the models MCI to MCn relating to 
each real camera on the conical projection model repre- 60 
sented in FIG. 5A in which each ray PM in the three-di- 
mensional space corresponds to a point m* in the two- 
dimensional space constituted by the source image 
plane of each camera. As these cameras are not ideal 
cameras but, in contrast, present distortions, this con- 65 
struction module 206 for the models MCI to MCn com- 
prises means for correcting distortions, with which 
means the point m* corresponding to a distorted source 
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image can be systematically replaced by a point m in the 
same two-dimensional space, which point corresponds 
to a corrected source image. All the means for correct- 
ing the distortions of the objectives may be known to 
those skilled in the art. 

The address computer 200 also comprises address 
computing means for determining the address Aq of this 
point m of the distorted source image Ij. 

These means are shown in the form of functional 
blocks in FIG. 6 and implement a method of computing 
the address Aq, described hereinafter and illustrated in 
FIG. 5B for the sake of clarity. 
VI.a/The method of computing the address. 

This method of computing the address Aq corre- 
sponding to an initial address Ao comprises several 
steps including at least a complex geometrical transform 
which is the product of several elementary transforms. 
This geometrical transform is based on the fact that, 
thanks to the module 206 described hereinbefore, there 
is now uniformity between each pixel m in a source 
image of a real camera and a light ray PM in the three- 
dimensional space passing through the view point P 
which is defined for the model of the camera. The local- 
ization of a pixel m in the image depends on the camera 
parameters relating to the definition of the model of this 
camera, while the corresponding light ray PM has a 
localization in the three-dimensional space which may 
be expressed with respect to the fixed landmark. 

The method of computing the address Aq on the basis 
of Ao includes the following steps: 
a first transform referred to as "inverse perspective 
transform" denoted Ho" 1 which indicates the di- 
rection of the light ray PM corresponding to the 
initial address Ao in the target image Io, 
searching the index j of the real camera Cj whose 
field of view comprises the light ray in question, 
and if several cameras are concerned, choosing an 
appropriate camera for a possible overlap of the 
fields of view, 
a second transform referred to as "direct perspective 
transform" denoted Hj which provides the address 
Aq in the image Ij of the selected real camera Cj, 
corresponding to the light ray PM. 
The two transforms, i.e. the inverse transform H" 1 
and the direct transform H thus depend on the defini- 
tion of the models of the cameras having a view point P 
of the pin-hole model. 

For realizing the complex geometrical transform H, 
an operation referred to as "perspective projection" J is 
first considered, which operation enables the user to 
pass from the point M of a scene, which is marked in the 
landmark (Px, Py, Pz), to the point m positioned in the 
image plane of a real camera Ci, the point m thus being 
marked in the individual orthonormal system related to 
this real camera, i.e. (Ojxj, Ojyj, Ojzj). 

With the "perspective projection" operation J, the 
marking of a point in a three-dimensional system is 
changed to a marking in a two-dimensional system by 
means of a mathematical method. This operation may 
be linearized by using the homogeneous coordinates 
with which the transform by a 4X4 matrix can be ex- 
pressed. 

The geometrical transform H referred to as "perspec- 
tive transform" is the product of several matrices with 
homogeneous coordinates: 
T is a matrix representing a translation, 
S is a matrix representing a change of the scale, 
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J is the "perspective projection" matrix referred to the blocks 22.1 to 22. n which are said second modules 
hereinbefore, for computing the geometrical transform and 

L is a change of origin, which receive the following information, 

R is a rotation in accordance with the angles <fcj, 0j, the selection signal from the switching module 205 

i/fj, or 4>a t Go, t|*o, as the case may be. 5 with the information relative to the source image 

The direct geometrical transform is thus expressed Ij » the case described by way of example where 

by. camera Cj has been selected, 

the model MCj of the real camera Cj from the 
H=T*S'J*l*r second construction module 206, 

10 the parameters <f>j, 9j, i/tf, zj of the real camera Cj 
and the inverse geometrical transform is expressed by: from the storage module 21 j (second storage 

means) in the case described. In the present case 
H -i =T -u s _wj-i» L -w R _i t j, e computation module 22.j (second means for 

computing the geometrical transform) computes 
This transform may be expressed in an approximate 15 the localization of the point m in the two-dimen- 

manner for an assembly of polynomial functions by sionsl space constituted by the plane of the 

using, for example the least-squares method. . source image Ij, on the basis of the localization 

The approximate transform thus obtained may then provided by the module 204 for storing the light 

be realized by means of a commercially available prod- ray PM, and this by means of the direct perspec- 

uct referenced TMC2302 of the firm of TRW. The 20 tive transform Hj described hereinbefore, 

address of this firm is: PO Box 2472, La Jolla, Calif. the blocks 23.1 to 23.n which are blocks for storing 
92038 (US). the addresses Aq of each point m. In the present 

VLb/Description of a particular implementation Of the case the address Aq of the point m searched will be 

address computer 200 available in the storage module 23.j. 

This implementation is illustrated by means of func- 25 If the switching module 205 had selected another 
tional blocks in FIG. 6, showing: source image, another computation module 22.1 to 22.n 

the block 201 which is the first construction module would have applied the appropriate direct perspective 
for the model of the virtual camera MCo, transform HI to Hn for localizing the point m in this 

the block 210 which is the first module for storing the other source image, 
parameters <f>o, 9o, ifro, zo of the virtual camera as 30 with reference to these operations effected in the 
effected by the user at the interface 2 of the control address computer 200, with reference to FIG. 3, this 
block | address computer 200 applies the address Aq to the 

the block 113 which is the generator for the addresses interpolator block 112 and the latter computes the cor- 
Ao of all the pixels m' of the target image lo, o responding luminance function F. The image lo may 
the block 220 which is a first module for computing 35 thus * reconstituted in the storage block 117 by. caus- 
the geometrical transform which receives the fol- in S a value of A func £ on * t0 f corres P ond to 

lowing information: ^ ch P ,xel m of the address Aa ^ !mage 15 now 

the address Ao of m' from the address generator ^^Lnce to FIG. 2, the image lo may subse- 

A , 1 , . - A . . . , f 40 quently be displayed, or stored in the means 10. 

the model MCo of ^ virtual camera from the Jf ^ ^ m ^ for 

construction module i201, because the source images overlap each other in a 

the . parameters <f,o, Go, *o zo of the virtual camera regiQn> ^ ^ 2Q5 conUdns 

from the fust storage module 210. means for choosing a single one of the two source im- 

In the three-dimensional space the computation block 45 ^ 
220 determines the position of the ray PM passing A* method of selecting one of the two images is to 
through the view point (P) and through the pixel m , dedde whether the seam betW een two images in the 
based on the localization of m' m the two-dimensional boundary region ^ constituted by a straight line which 
space la To this end, this computation block 220 ap- substantially passes through the middle of the overlap 
plies the inverse perspective transform Ho" 1 as de- 50 zone -j^ method j s applicable to cameras which are 
scribed hereinbefore, juxtaposed horizontally, as in FIG. 1, with vertical 

the block 204 which is a module for storing the posi- seams. 

tion of the light ray PM in the three-dimensional Those skil ] e(1 in the ^ win ^ able to choose all kinds 

space, which position is provided by the computa- 0 f searnS( f or example, seams which are arranged radi- 

tion block 220, 55 ally on conies, 

the selection block 205 which is a switching module j t j s t0 be notec j that what has been stated for coding 

and which, based on the localized ray PM, selects the pixels at a grey level may also apply to coding a 

the real camera Cj and provides the corresponding co ] or image. The coding is simply effected for a larger 

source image Ij in which the searched point m is number of bits. 

present, 60 In order that no seam is noticeable in the target im- 

the block 206 which is the second module for con- a ge, it should be noted that it is necessary to take several 
structing the models MCI to MCn of the real cam- measures: 

eras with means for correcting the distortions being precise calibration of the fixed cameras, 
incorporated, correction of the results by applying feedback, fol- 

the blocks 21.1 to 21.n which are the storage blocks 65 lowed by a possible modification of the parameters 
referred to as second means for storing the parame- of the fixed camera models until a perfect juxtapo- 

ters <f>l to <fm, Gl to 6n, *|*1 to ifm, zl to zn, respec- sition contributing to a target image without any 

tively, of the real cameras, faults is obtained. 



Page 15 (MLee3, 12/28/2000, 



EAST Version: 1.01.0015) 



17 



5,444,478 



18 



This operation, or initialization, should be an integral 
part of the calibrating operation. 

It is also to be noted that distortion corrections are to 
be performed on each source image for the construction 
of the models MCI to MCn. Devices capable of realiz- 5 
ing these operations are commercially available. Verifi- 
cation of the perfect operation of the distortion correct- 
ers used is also an integral part of the calibration. 

The different real cameras may have different color 
or grey level intensity response curves. During calibra- 10 
tion, it is necessary to perform a correction of these 
individual responses so as to obtain a uniform virtual 
image. 

The user interface 2 may be controlled automatically 
or manually. In one implementation, this interface may 15 
be a joystick. 

We claim: 

1. A method of processing images for constructing a 
target image (Io) from adjacent images having a fixed 
frame line and referred to as source images (II. • • • > Ii> 20 
Ij, . . . , In), said source and target images having sub- 
stantially common view points, characterized in that the 
method comprises the steps of: 

(a) digitizing said source and target images; 

(b) determining a substantially common view point 25 
(P) to said source and target images, and a fixed 
orthonormal landmark (Px, Py, Pz) originated at 
said common view point (P); 

(c) generating, pixel by pixel, an address, for each 
pixel, in the target image (Io), so as to entirely 30 
cover said target image (Io); 

(d) calculating, on the basis of an address (Ao) of a 
current pixel referred to as an initial pixel (m') in 
the target image (Io), an orientation, in said fixed 
landmark, of a straight light ray (PM) passing 35 
through said initial pixel (m') and through said 
common view point; 

(e) selecting a source image (Ij) traversed by said 
straight light ray (PM); 

(f) calculating, from said orientation of the straight 40 
light ray (PM), an address (Aq) of a distortion 
corrected point (m), in said selected source image 
(Ij), said distortion corrected point (m) correspond- 
ing to said initial pixel (m'); 

(g) calculating a luminance value (F) at said distor- 45 
tion correct point m in said source image Ij; 

(h) assigning the luminance value (F) of said distor- 
tion corrected point (m) to the initial pixel (m t ) in 
the target image (Io); 

(i) repeating steps (d)-(h) for each pixel of the target 50 
image (Io), until all pixels of the target image (Io) 
have been processed. 

2. A method as claimed in claim 1, characterized in 
that said method comprises calibration steps including: 

storing parameters of the source images, including 55 
scale factors (zl, . . . , zn) and orientation angles 
constituted by azimuth angles (61, . . . , 8n), angles 
of sight (4>1, .... <J>n) and angles of rotation Ofrl, . 
. . , ifm) of source images optical axis, said angles 
being defined in said fixed orthonormal landmark 60 
(Px, Py, Pz); 

constructing real camera models for providing said 
source images with said view point which is sub- 
stantially common with the origin of said land- 
mark, and for correcting aberrations in said source 65 
images; 

storing target image parameters, including scale fac- 
tor (zo) and orientation angles constituted by an 



azimuth angle (Bo), an angle of sight (t|>o) and an 
angle of rotation (\po) of the target image optical 
axis, said angles being defined in said fixed ortho- 
normal landmark (Px, Py, Pz); and 
constructing a virtual camera model for providing 
said target image with said view point which is 
substantially common with that of the source im- 
ages. 

3. A method as claimed in claim 2, characterized in 
that 

the step of determining the position of the straight 
light ray (PM) in said landmark comprises a geo- 
metrical transform referred to as "inverse perspec- 
tive transform" (Ho -1 ) which takes the virtual 
camera model into account; 

the step of determining the address (Aq) of said dis- 
tortion corrected point (m) corresponding to said 
initial pixel (m*) comprises perspective geometrical 
transforms (Hl-Hn) referred to as "direct perspec- 
tive transforms", which, in accordance with said 
selected source image Ij, take into account the 
respective real camera model corresponding to said 
selected source image. 

4. A method as claimed in claim 3, characterized in 
that the step of 

calculating a luminance value (F) at said distortion 
corrected point (m) in said selected source image 
(Ij) includes performing an interpolation for com- 
puting a most probable value of a luminance func- 
tion (F) at the address (Aq) in the source image (Ij). 

5. A method as claimed in claim 1, characterized in 
that the step of 

calculating a luminance value (F) at said distortion 
corrected point (m) in said selected source image 
(Ij) includes performing an interpolation for com- 
puting a most probable value of a luminance func- 
tion (F) at the address (Aq) in the source image (Ij). 

6. An image processing device comprising: 2 

a system of n fixed real cameras (CI, . . . , Ci, Cj, . . 
. , Cn) arranged in such a way that their individual 
fields of view merge so as to form a single wide-an- 
gle field of view for observation of a panoramic 
scene, said real cameras providing adjacent images 
referred to as source images; 

an image construction system simulating a mobile, 
virtual camera (Co) continuously scanning the 
panoramic scene so as to form a sub-image referred 
to as target image (Io) corresponding to a selected 
section of the wide-angle field of view and con- 
structed from said source images (II, . , . , Ii, Ij, . . 
. , In) furnished by the n real cameras, character- 
ized in that said image processing device com- 
prises: 

means for digitizing said source and target images; 

calibration means for determining a substantially 
common view point (P) to said images, and a fixed 
orthonormal landmark (Px, Py, Pz) originated at 
said common view point; 

an address generator for generating, pixel by pixel, 
respective addresses for the pixels of said target 
image (Io) so as to cover the entire target image 
Go); 

an address computer for calculating, on the basis of 
an address (Ao) of a current pixel referred to as 
initial pixel (m') in the target image (Io), an orienta- 
tion, in said fixed landmark, of a straight light ray 
(PM) passing through said initial pixel (m') and 
through said common view point (P) , selecting a 
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source image (Ij) traversed by said straight light 
ray (PM), calculating, from said orientation of said 
straight light ray (PM), an address (Aq) of a distor- 
tion corrected point (m), in said selected source 
image (Ij), said distortion corrected point (m) cor- 5 
responding to said initial point (m'); and 
means for determining a luminance value (F) at said 
distortion corrected point (in), and assigning said 
luminance value to said initial point (m') . 

7. A device as claimed in claim 6, characterized in 10 
that the calibration means further comprises: 

first storage means for storing the parameters relating 
to said virtual camera for supplying the address 
computer with a scale factor (zo) and orientation 
angles of the optical axis of said virtual camera 15 
(Co) in said fixed orthonormal landmark (Px, Py, 
Pz) which is independent of the cameras, said ori- 
entation angles being constituted by an azimuth 
angle (8o), an angle of sight (<f>o) and an angle of 
rotation (^o); 20 

second storage means for storing parameters relating 
to said real cameras (Cl-Cn) for supplying said 
address computer with scale factor (zl-zn) and 
with orientation angles of an optical axis of each 
real camera (CI, . . . , Ci, Cj, Cn), said orientation 25 
angles being constituted by azimuth angles 
(01-Gn), angles of sight (<f)l-<^n) and angles of 
rotation (ifrl-xfm) defined in said fixed landmark. 

8. A device as claimed in claim 7, characterized in 
that the address computer comprises: 30 

first construction means for constructing a model 
(MCo) of the virtual camera with a projection via 
the view point P; and 

second construction means for constructing models 
(MCl-MCn) of the real cameras with a projection 35 
via the view point P and with corrections of distor- 
tions. 

9. A device as claimed in claim 8, characterized in . 
that the address computer comprises: 

first means for computing a geometrical transform, 40 
referred to as "inverse perspective transform" 
(Ho -1 )* to said initial pixel (m') at an address (Ao) 
of the image (Io) of the virtual camera (Co), in said 
inverse perspective transform, the model of the 
virtual camera (MCo) provided by said first con- 45 
struction means and the parameters constituted by 
the azimuth angle (Go), the angle of sight (<(>o), the 
angle of rotation (xpo) and the scale factor (zo) of 
said virtual camera provided by said first storage 
means being taken into account for determining, on 50 
the basis of said inverse perspective transform 
(Ho -1 ), the positioning, in said landmark, of said 
straight light ray passing through said initial pixel 
(m') and through the view point (P); 

means for storing the position of said straight light 55 
ray obtained by the inverse perspective transform 
(Ho- 1 ); 

selection means for selecting a source image (Il-In) 

traversed by said straight light ray; 
second means for computing a geometrical transform, 60 

referred to as "direct perspective transform" 
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(Hl-Hn), to said straight light ray in said landmark, 
said direct perspective transform, the models of the 
real cameras provided by the second construction 
means, the parameters constituted by the azimuth 
angles (01-0n), the angles of sight (<f>l-<f>n), the 
angles of rotation (Bl-Bn) and the scale factors 
(zl-zn) of the respective real camera (Cl-Cn) cor- 
responding to said selected source image provided 
by said second storage means being taken into ac- 
count; and 

storage means for supplying, on the basis of said di- 
rect perspective transform (Hl-Hn), an address 
(Aq) in said selected source image (Il-In) which 
corresponds to said straight light ray and thus to 
said initial pixel at the address (Ao) in the target 
image (Io). 

10. A device as claimed in claim 9, characterized in 
that the means for determining the luminance comprise: 

an interpolator for computing a most probable value 
(F) of a luminance function at the address (Aq) 
found by the address computer in said selected 
source image furnished by the selection means; and 

third storage means for assigning said computed lumi- 
nance value (F) corresponding to the address (Aq) 
found in said selected source image to the initial 
pixel in the target image (Io) at the address (Ao) 
furnished by said address generator. 

11. A device as claimed in claim 10, characterized in 
that said device further comprises a display system with 
a screen for displaying the target image (Io) in real time 
on said screen. 

12. A device as claimed in claim 10, characterized in 
that the device further comprises a recording system for 
recording the target image (Io). 

13. A device as claimed in claim 10, characterized in 
that the system for constructing the target image (Io) 
also comprises: 

an interface for enabling a user to define said parame- 
ters of the virtual camera (Co), said parameters 
including the scale factor (zo) and the orientation 
of the optical axis (6o, <J>o, ^o). 

14. A device as claimed in claim 13, characterized in 
that the user interface is controlled automatically or 
manually. 

15. A device as claimed in claim 6, characterized in 
that the system for constructing the target image (Io) 
also comprises: 

an interface for enabling a user to define parameters 
for the virtual camera (Co), said parameters includ- 
ing a scale factor (zo) and orientation angles (6o, 
<J>o, ijfo) for the optical axis defined in said fixed 
landmark of said virtual camera. 

16. A device as claimed in claim 6, characterized in 
that said device further comprises a display system with 
a screen for displaying the target image (Io) in real time 
on said screen. 

17. A device as claimed in claim 6, characterized in 

that the device further comprises a recording system for 

recording the target image (Io). 

***** 
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